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Summary 

The Network Control Center (NCC) Project introduced the concept of total 
quality management (TQM) in mid-1990. The CSC project team established a 
program which focused on continuous process improvement in software 
development methodology and consistent deliveries of high quality software 
products for the NCC. The vision of the TQM program was to produce error free 
software. Specific goals were established to allow continuing assessment of 
the progress toward meeting the overall quality objectives. The total quality 
environment, now a part of the NCC Project culture, has become the foundation 
for continuous process improvement and has resulted In the consistent delivery 
of quality software products over the last three years. 


Background 


The Network Control Center (NCC) Project has had a long history of developing 
and maintaining software to assure that the system stays abreast of the 
changing needs of the Space Network. The NCC must schedule, control, and 
monitor the real time activities associated with providing communications 
support to spacecraft requiring the capabilities of the Tracking and Data 
Relay Satellite System (TDRSS). 

In 1989, a major initiative was underway to modernize the White Sands Complex 
by establishing a Second TDRS Ground Terminal and upgrading the existing 
terminal to handle the projected workload for the late 1990s. These changes in 
the Space Network necessitated major changes in the NCC to assure that control 
center software would be compatible with the ground terminal environment. 
This development activity was designated as NCC Block 3 and the project was 
tasked to develop software that would be delivered in three releases. The 
first release would accommodate the changes necessitated by the Second TDRS 
Ground Terminal (STGT) . The software product would have to operate with 
formats and functionality that could support the current ground terminal 
environment and the STGT. The second release was to be designed to handle the 
STGT and the White Sands Ground Terminal Upgrade (WSGTU) . The final release 
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had the objective of providing the capability to schedule, control, and 
monitor a four or more TDRS constellation. The planned completion of the NCC 
Block 3 software upgrades was scheduled for 1995. 

In parallel with the challenge to develop the Block 3 software, the NCC 
Project had to provide software enhancements to the operational baseline to 
address known system deficiencies and planned hardware upgrades. These 
activities were to be accomplished under the NCC Block 1 or maintenance 
effort. Maintenance deliveries were planned for each fiscal year and all 
operational baseline changes were integrated into the development baselines. 

All development planning actions had to take into consideration the impact of 
the ongoing maintenance tasks. 

Management Realignment 

Within the context of this phased software development effort, the concept of 
total quality management was introduced in mid-1990. During the same time 
frame Goddard Space Flight Center (GSFC) management aligned the NCC 
Project software development efforts within the Flight Dynamics Division (FDD) 
from the traditional Networks Division (ND) in the Mission Operations & Data 
Systems Directorate. This action was taken because it was determined that the 
appropriate project management structure had to be in place for the NCC Block 
3 software development objectives to be successfully met. This structure was 
not in place in the Networks Division nor was there sufficient expertise to 
establish the required organization. Thus, a project organization was 
established within the FDD to manage the CSC SEAS NCC Project Block 3 software 
development effort. 

In concert with the management changes made on the GSFC staff, CSC assigned a 
new project manager and deputy project manager to lead the NCC Project. This 
new team established, within the NCC Project, a total quality program which 
focused on continuous process improvements in the software development 
methodology and consistent deliveries of high quality software products for 
the NCC. The vision of the total quality program was to establish and maintain 
an environment in which software products could be developed and delivered 
error free. To accomplish this goal, the project had to emphasize its 
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understanding of customer needs, appreciate the value of continuous process 
improvement, and recognize the importance of team building and effective 
communications. 

Several measures of merit were established to allow for continuous assessment 
of the progress toward meeting the overall quality objective. These measures 
were categorized as the "FOUR P’s"; Product; Process; Performance; and 
Participation. The "Product" criteria was measured in terms of the quality of 
delivered software and the project's ability to deliver on schedule and within 
budget. The critical quality measures were related to the error rates during 
system and acceptance testing. The "Process" criteria was measured in terms 
of process improvements implemented which contributed to product quality 
and/or cost avoidance. These accomplishments are documented in success stories 
and task improvement initiatives. The "Performance" criteria was measured by 
the monthly award fee evaluations. The performance target was to achieve at 
least 50% plus evaluations during a given fiscal year. The "Participation 
measure was the percentage of project personnel who participated in TQM 
activities. The target was to achieve and sustain a 75% participation rate. 
Measurements against these goals are accomplished on a monthly basis and 
feedback is provided to project personnel through newsletters and all-hands 
meetings * 

The NCC Project TQM environment provided the framework for continuously 
improving the quality of all NCC products and services to meet established 
goals and objectives. The TQM environment established a highly participative 
management methodology that complemented the NCC organizational structure. 
The focus of the methodology was a long-term commitment to improving the 
quality of products and services. This methodology used a hierarchy of NCC 
management-chartered committees to improve the quality of NCC processes and 
products. These committees accomplished this by identifying, assessing, 
measuring, analyzing, and documenting processes, issues, and products and 
recommending improvement action plans to management. 

The TQM program was implemented through the NCC Quality Management Board 
(QMB) , designated permanent committees and selected ad hoc committees. The 
five permanent committees include the Suggestion Analysis and Resolution Group 
(SARG), the Customer Satisfaction Committee (CSAT), the Worklife and 
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Communications Committee (WLCC) , the Process Improvement Committee (PIC), and 
the Awards and Recognition Committee (ARC). The SARG was chartered to analyze 
suggestions for improving products and processes. Following the analysis, a 
recommendation for implementation would be provided. The CSAT was chartered 
to establish consistent and meaningful mechanisms for obtaining client 
feedback and measuring customer satisfaction. The WLCC was chartered to 
develop, implement, and facilitate an effective NCC communications program. 
The PIC was chartered to focus on methods to improve technical processes. 
Many of the ad hoc Process Action Teams (PATs) evolved from PIC initiatives to 
update or modify existing procedures. The ARC was chartered to develop, 
implement, and facilitate the NCC recognition program. Their work was the 
basis for the monthly recognition program at all-hands meetings and the annual 
NCC DATUM award to the project employee of the year. 

The Total Quality Impact 

Given the scope of the tasks that the project had to accomplish, the 
organizational environment (i.e. the FDD GSFC management team), and the newly 
established TQM environment for the SEAS NCC project team, a look at the 
results and on-going activity provides an interesting perspective on how a 
large software development project can succeed. 

One of the most important contributors to success was responsiveness. This 
was demonstrated in the initial transition of the project to the FDD. Close 
coordination and communication among all parties, intense effort, and a strong 
desire to succeed paved the way to assure a smooth transition took place. 
Responding to five new task assignments which mapped to the development, 
maintenance, system engineering, system testing, and security requirements, 
the NCC project team replanned all the work in less than one month. To 
expedite task planning, CSC management and technical personnel worked closely 
with GSFC managers to familiarize them with the NCC system architecture, 
software environment, and system functionality. Frequent meetings between the 
project task managers and their GSFC counterparts ensured that the technical 
effort mapped correctly to the new task structure. 

During the transition, project personnel were preparing to meet two critical 
milestones. On August 7, 1990, the CDR for Block 3, Release 1 was held 
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successfully. Related deliveries included the detailed design document, 
operations concept document, quality assurance plan, system test plan, and 
configuration management plan - collectively, more than 4000 pages. This 
event concluded the detailed design phase and clearly demonstrated the 
readiness for software implementation. The relatively small number of Review 
Item Disposition (RIDs) (31), none of which affected the overall design 
direction, testifies to the high quality of the design. The second event was 
the delivery on September 17 of the FY90 maintenance release to acceptance 
testing 2 weeks early. The quality of the software was evident from the low 
number of problem reports (11) identified and corrected during system testing. 
This release, designated 90.1, completed acceptance test one month earlier 
than scheduled. Release 90.1 was the first delivery to operations in which the 
label "error-free" could be applied since there were no errors associated with 
new or modified functions. 

As the project stabilized in the FDD environment, significant milestones 
continued to be met. The FY91 maintenance release, 91.1, completed system 
testing as scheduled. The upgrade of the Communications and Control Segment 
(CCS) VAX to an 8550 processor configuration was system tested and integrated 
into the operational configuration. The Block 3, Release 1, Build 1 activity 
completed integration testing on schedule and made significant progress in 
implementing Build 2 during the spring of 1991. Release 91.1 was delivered to 
operations during the summer of 1991. This release consisted of operating 
system upgrades to both the UNISYS and DEC processors. These upgrades were 
implemented as a risk mitigation action for Release 1 to minimize the amount 
of change occurring with a planned development release. The lesson learned was 
that operating system changes in a real time system environment can have 
unexpected results as was evident during transition to operations. Both 
timing and performance deficiencies were uncovered which required quick fix 
actions to maintain system stability. The final delivery for release 91.1 was 
accomplished in November 1991. 

In parallel with the maintenance activity, the Block 3, Release 2 CDR was 
successfully held in October 1991, while system testing of the Release 1 final 
build was progressing as planned. As another risk mitigation activity, the 
system test team performed limited interface testing with STGT while it was at 
the vendor's site. These tests proved invaluable in uncovering problems early 
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and assigning responsibility for resolution. This activity contributed to the 
establishment of the interface incident reporting (HR) process which is now 
in place to document and resolve interface incompatibilities within the Space 
Network. Block 3, Release 1/ was delivered to acceptance test in March 1992 
on schedule and within budget. The release was over 100,000 Delivered Source 
Instruction (DSI) and a relatively small number of problem reports were 
identified during acceptance testing. After a successful ORR on August 27, 
1992, the Block 3, Release 1 (Release 92.1) was successfully transitioned to 
operations on September 25, 1992. 

To achieve this successful delivery over a two year period, several 
significant process improvements were put in place, resulting in a more 
disciplined software development environment and a significantly higher 
quality product. During implementation, all changes were subjected to a 
rigorous design, code, and unit test inspection and certification process. 
The configuration management team implemented procedures that reduced baseline 
build errors by over 50 percent. The system test team developed more detailed 
functional and regression test procedures that reduced the number of errors 
found by the acceptance test team by over 25 percent of that projected for a 
release of this Bize. Overall, the software engineering process improvements 
implemented during the Release 1 development cycle have resulted in a 
capability for the NCC to be compatible with all Space Network elements and to 
operate more reliably in the STGT era. 

Block 3, Releases 2 and 3 are still under development and progressing as 
planned. Release 93.1, the FY93 maintenance release, was successfully 
delivered to operations on August 25, 1993 with the highest quality measure 
for any NCC delivered product. 

The Transition Back To Networks Division 

After two and a half years, the software development effort was transitioned 
back to the Networks Division. The lessons learned from the Flight Dynamics 
software management experience were an integral part of the successful 
transition back to the Networks Division. A comparable GSFC management team 
was established to work with the Computer Sciences Corporation (CSC) NCC 
Project management team. 
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In order to accomplish a successful transition, a plan was developed in which 
a period of 9 months was allocated for transition activities. The first step 
was to establish a comparable GSFC technical management team in the Networks 
Division. This team was carefully selected and in April 1992 they began 
attending the weekly task meetings. This provided a mechanism for the new 
team to become familiar with the technical issues, the development 
methodology, and the technical management roles and responsibilities. During 
the first 6 months of the transition period, the FDD team was in the decision 
making role for the project. During the last 3 months, the ND team took the 
lead while the FDD team assumed the role of adviser and observer. At the 
beginning of 1993, the transition was completed without impact on any critical 
NCC milestones. 

Kev Lessons Learned 

The key lessons learned were in four areas: (1) commitment at all levels in 
the project - both CSC and GSFC; (2) requirements control and effective change 
management vital; (3) product focus as well as process focuB is key to 
success; and (4) consistent measurement of critical success factors. 

The management commitment to the NCC Project was visible and sincere on the 
part of GSFC and CSC. In the spring of 1990, CSC not only selected a new 
management team for the project but also established periodic meetings between 
senior MO&DSD personnel and CSC corporate personnel. These meetings were held 
quarterly and provided the forum for discussion of issues which might impact 
the project’s objectives. The participants at these meetings included the 
Director and/or Deputy Director of MO&DSD, the CSC Systems Group President, 
the Systems Sciences Division President, the SEAS Program Manager, and the 
SEAS Deputy Program manager. This level of commitment assured that the 
appropriate level of attention and resources would be applied should an issue 
arise which could potentially impact the project. The periodic meetings 
continued through the transition of the project back to the Networks Division 
and will be convened in the future as required. 
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Requirements control and change management were vital to the success of the 
project. The management decision to separate the software development 
activities from the requirements definition functions was critical to 
requirements control. The GSFC FDD management team had the responsibility to 
focus on the software development progress and control the growth of the 
system. As new or changed requirements were identified by the ND NCC Project 
Office^ an impact assessment was performed and a conscious decision was made 
to consider adjusting established milestones and resources to accommodate new 
requirements. In addition to the external pressures to change requirements/ 
there were internal proposed changes that were identified during the design 
and development phases of the implementation. To control these changes/ 
processes were put in place for both technical and management reviews of all 
proposed changes through the Technical Review Board (TRB) and Configuration 
Review Board (CRB). These internal project reviews are an integral part of 
the change management process and provided a more rigorous approach to 
managing the scope and size of the software effort. 

With the establishment of the Total Quality Management Program, there was 
significant emphasis on continuous process improvement. However, in parallel 
with that emphasis, there was need to keep the focus on product improvement as 
well. This was accomplished by establishing a clear cut vision which focused 
on the product and challenged project personnel to develop and deliver error- 
free software. By having this focus, all impacts of process improvement 
activities could be measured either directly or indirectly with the quality of 
the delivered release. The result of the product focus can be seen in Figure 
1 which shows the error rate of the major releases over the last four years. 
The results of the process improvements are readily seen in the trends showing 
a steady decrease in the error rate during system and acceptance testing. 

The final area where lessons were learned was in the establishment of a 
measurement program and sustaining it to provide visibility into the project's 
progress toward meeting targeted objectives. The "Product” measure as stated 
above was the error rates experienced during the test phases. This is 
illustrated in Figure 1. The impact of the quality initiatives is clearly seen 
in the error rate in releases 92.1 and 93.1 which were .24 and .19 errors/KDSI 
respectively. The "Performance" measure is shown in Figure 2 where the goal 
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of 50% plus evaluations is on target through fiscal year 1993. The "Process" 
measure of success stories is demonstrated by the fact that during FY1993, 
there have been 18 success stories written by project members with a total 
cost avoidance value of $ 387,000 . The final measure was "Participation". 
During the current fiscal year, participation has exceeded 85% of project 
personnel. These measures have been used for management assessment of the TQM 
program, but these initiatives have had a much deeper impact on the project. 
The total quality environment is now a part of the NCC Project culture. It 
remains the foundation for continuous process improvement on the project and 
is the basis for the consistent quality of all NCC software products. 
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